Flash optimized, log-structured layer of a file system

ABSTRACT

A flash-optimized, log-structured layer of a file system of a storage input/output (I/O) stack executes on one or more nodes of a cluster. The log-structured layer of the file system provides sequential storage of data and metadata (i.e., a log-structured layout) on solid state drives (SSDs) of storage arrays in the cluster to reduce write amplification, while leveraging variable compression and variable length data features of the storage I/O stack. The data may be organized as an arbitrary number of variable-length extents of one or more host-visible logical units (LUNs) served by the nodes. The metadata may include mappings from host-visible logical block address ranges (i.e., offset ranges) of a LUN to extent keys, as well as mappings of the extent keys to SSD storage locations of the extents. The storage location of an extent on SSD is effectively “virtualized” by its mapped extent key (i.e., extent store layer mappings) such that relocation of the extent on SSD does require update to volume layer metadata (i.e., the extent key sufficiently identifies the extent).

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 14/150,717, entitled Flash Optimized, Log-Structured Layer of aFile System, filed on Jan. 8, 2014 by Rajesh Sundaram, et al., and isrelated to U.S. patent application Ser. No. 14/160,991, filed on Jan.22, 2014, entitled Flash Optimized, Log-Structured Layer of a FileSystem, by Rajesh Sundaram, et al., now issued as U.S. Pat. No.8,880,788 on Nov. 4, 2014, which applications are hereby incorporated byreference.

BACKGROUND

Technical Field

The present disclosure relates to storage systems and, morespecifically, to a flash optimized, log-structured layer of a filesystem of one or more storage systems of a cluster.

Background Information

A storage system typically includes one or more storage devices, such assolid state drives (SSDs) embodied as flash storage devices, into whichinformation may be entered, and from which the information may beobtained, as desired. The storage system may implement a high-levelmodule, such as a file system, to logically organize the informationstored on the devices as storage containers, such as files or logicalunits (LUNs). Each storage container may be implemented as a set of datastructures, such as data blocks that store data for the storagecontainers and metadata blocks that describe the data of the storagecontainers. For example, the metadata may describe, e.g., identify,storage locations on the devices for the data. In addition, the metadatamay contain copies of a reference to a storage location for the data(i.e., many-to-one), thereby requiring updates to each copy of thereference when the location of the data changes, e.g., a “cleaning”process. This contributes significantly to write amplification as wellas to system complexity (i.e., tracking the references to be updated).

Some types of SSDs, especially those with NAND flash components, may ormay not include an internal controller (i.e., inaccessible to a user ofthe SSD) that moves valid data from old locations to new locations amongthose components at the granularity of a page (e.g., 8 Kbytes) and thenonly to previously-erased pages. Thereafter, the old locations where thepages were stored are freed, i.e., the pages are marked for deletion (oras invalid). Typically, the pages are erased exclusively in blocks of 32or more pages (i.e., 256 KB or more). This process is generally referredto as garbage collection and results in substantial write amplificationin the system.

In addition, the “on-disk” layout of the data structures in the storagecontainers (i.e., on the SSDs) may create a plurality of odd-shapedrandom “hole” (i.e., deleted data) fragments adjacent to data. Thisfragmented data (i.e., data with interposed holes) may not facilitatenatural alignment boundaries for Redundant Array of Independent Disk(RAID) configurations, thus raising problematic RAID implications. Forexample, if an attempt is made to write data into the odd-shapedfragments, it may be difficult to achieve good RAID stripe efficiencybecause partial stripes may be written, causing increased writeamplification due to increased parity overhead.

Yet another source of write amplification in the system may involveRAID-related operations. Assume a dual parity RAID implementation thatmay include a plurality of data SSDs and two parity SSDs. A random writeoperation that stores write data on a data SSD of a RAID stripe mayresult in a plurality of read-modify-write (RMW) operations that, e.g.,updates the data SSD with write data and updates the two parity SSDswith parity information after reading a portion of the write data and/orparity information. Such RAID-related operations results in asubstantial amount of write amplification to the system.

Therefore, it is desirable to provide a file system that reduces varioussources of write amplification from a storage system, wherein thesources of write amplification include, inter alia, 1) storage locationreference updates; 2) internal SSD garbage collection; 3) partial RAIDstripe operations from fragmented data; and 4) RMW operations from RAIDorganizations of data and parity.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and further advantages of the embodiments herein may be betterunderstood by referring to the following description in conjunction withthe accompanying drawings in which like reference numerals indicateidentically or functionally similar elements, of which:

FIG. 1 is a block diagram of a plurality of nodes interconnected as acluster;

FIG. 2 is a block diagram of a node;

FIG. 3 is a block diagram of a storage input/output (I/O) stack of thenode;

FIG. 4 illustrates a write path of the storage I/O stack;

FIG. 5 illustrates a read path of the storage I/O stack;

FIG. 6 illustrates a layered file system;

FIG. 7a illustrates segment cleaning by a log-structured layer of thefile system;

FIG. 7b illustrates hot and cold segments used by the log-structuredlayer of the file system; and

FIG. 8 illustrates a RAID stripe formed by the log-structured layer ofthe file system.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The embodiments described herein are directed to a flash-optimized,log-structured layer of a file system of a storage input/output (I/O)stack executing on one or more nodes of a cluster. As described herein,the log-structured layer provides sequential storage of data andmetadata (i.e., a log-structured layout) on solid state drives (SSDs) ofstorage arrays in the cluster to reduce write amplification, whileleveraging variable compression and variable length data features of thestorage I/O stack. The data may be organized as an arbitrary number ofvariable-length extents of one or more host-visible logical units (LUNs)served by the nodes. The metadata may include mappings from host-visiblelogical block address ranges (i.e., offset ranges) of a LUN to extentkeys, as well as mappings of the extent keys to SSD storage locations ofthe extents.

In an embodiment, an extent store layer of the file system performs andmaintains the mappings of the extent keys to SSD storage locations,while a volume layer of the file system performs and maintains themappings of the LUN offset ranges to the extent keys. Illustratively,the volume layer cooperates with the extent store layer to provide alevel of indirection that facilitates efficient log-structured layout ofextents by the extent store layer. That is, the extent key mappingsmaintained by the volume layer allow relocation of the extents on SSDduring, e.g., segment cleaning, without update to the volume layermappings. Accordingly, the storage location of an extent on SSD iseffectively “virtualized” by its mapped extent key (i.e., extent storelayer mappings) such that relocation of the extent on SSD does notrequire update to volume layer metadata (i.e., the extent keysufficiently identifies the extent). The virtualization of storagelocations permits cleaning processes to occur in the extent store layerwithout update to volume layer metadata, thereby substantially reducingwrite amplification.

In an embodiment, functions of the log-structured layer of the filesystem, such as write allocation and flash device (i.e., SSD)management, are performed and maintained by the extent store layer.Flash device management may include segment cleaning to illustrativelyclean one or more selected regions or segments that indirectly map toSSDs. To clean a selected segment, the extent store layer may moveextents of the segment that contain valid data to one or more differentsegments so as to free the selected segment. Once the segment isdesignated freed, additional extents may be written as a sequence ofwrite operations to the segment in accordance with the log-structuredlayout so as to optimize storage performance of the flash storagedevices (e.g., to reduce garbage collection within the SSD).Illustratively, the log-structured layer of the file system may writethe extents as a sequence of contiguous range write operations to theentire segment with temporal locality so as to reduce data relocation(i.e., internal flash block management) that may occur within the SSDsas a result of the write operations. It should be noted that thesequence of write operations need not be in a specific order (e.g., inascending range), but that the operations be of contiguous ranges andwritten with temporal locality.

Write allocation may include gathering of the variable-length extents toform one or more stripes across SSDs of one or more RAID groups. In anembodiment, the RAID layer may manage parity computations and topologyinformation used for placement of the extents on the SSDs of each RAIDgroup. To that end, the RAID layer may cooperate with the extent storelayer to organize the extents as stripes within the RAID groups.Illustratively, the extent store layer may gather the extents to formone or more full stripes that may be written to a free segment such thata single stripe write operation may span all SSDs in a RAID group. Theextent store layer may also cooperate with the RAID layer to pack eachstripe as a full stripe of variable-length extents. Once the stripe iscomplete, the RAID layer may pass the full stripe of extents to astorage layer of the storage I/O stack for storage on the SSDs. Bywriting a full stripe (i.e., data and parity) to the free segment, thelog-structured layer of the file system avoids the cost of parityupdates (i.e., read-modify-write of existing data in a stripe) andspreads the read operation load across the SSDs.

Advantageously, the volume and extent store layer file systems maycooperate to store the extents on SSD in a log-structured manner that isflash friendly with respect to random write operations and that reduceswrite amplification (e.g., particularly RAID-related writeamplification).

DESCRIPTION

Storage Cluster

FIG. 1 is a block diagram of a plurality of nodes 200 interconnected asa cluster 100 and configured to provide storage service relating to theorganization of information on storage devices. The nodes 200 may beinterconnected by a cluster interconnect fabric 110 and includefunctional components that cooperate to provide a distributed storagearchitecture of the cluster 100, which may be deployed in a storage areanetwork (SAN). As described herein, the components of each node 200include hardware and software functionality that enable the node toconnect to one or more hosts 120 over a computer network 130, as well asto one or more storage arrays 150 of storage devices over a storageinterconnect 140, to thereby render the storage service in accordancewith the distributed storage architecture.

Each host 120 may be embodied as a general-purpose computer configuredto interact with any node 200 in accordance with a client/server modelof information delivery. That is, the client (host) may request theservices of the node, and the node may return the results of theservices requested by the host, by exchanging packets over the network130. The host may issue packets including file-based access protocols,such as the Network File System (NFS) protocol over the TransmissionControl Protocol/Internet Protocol (TCP/IP), when accessing informationon the node in the form of storage containers such as files anddirectories. However, in an embodiment, the host 120 illustrativelyissues packets including block-based access protocols, such as the SmallComputer Systems Interface (SCSI) protocol encapsulated over TCP (iSCSI)and SCSI encapsulated over FC (FCP), when accessing information in theform of storage containers such as logical units (LUNs). Notably, any ofthe nodes 200 may service a request directed to a storage containerstored on the cluster 100.

FIG. 2 is a block diagram of a node 200 that is illustratively embodiedas a storage system having one or more central processing units (CPUs)210 coupled to a memory 220 via a memory bus 215. The CPU 210 is alsocoupled to a network adapter 230, storage controllers 240, a clusterinterconnect interface 250, and a non-volatile random access memory(NVRAM 280) via a system interconnect 270. The network adapter 230 mayinclude one or more ports adapted to couple the node 200 to the host(s)120 over computer network 130, which may include point-to-point links,wide area networks, virtual private networks implemented over a publicnetwork (Internet) or a local area network. The network adapter 230 thusincludes the mechanical, electrical and signaling circuitry needed toconnect the node to the network 130, which illustratively embodies anEthernet or Fibre Channel (FC) network.

The memory 220 may include memory locations that are addressable by theCPU 210 for storing software programs and data structures associatedwith the embodiments described herein. The CPU 210 may, in turn, includeprocessing elements and/or logic circuitry configured to execute thesoftware programs, such as a storage input/output (I/O) stack 300, andmanipulate the data structures. Illustratively, the storage I/O stack300 may be implemented as a set of user mode processes that may bedecomposed into a plurality of threads. An operating system kernel 224,portions of which are typically resident in memory 220 (in-core) andexecuted by the processing elements (i.e., CPU 210), functionallyorganizes the node by, inter alia, invoking operations in support of thestorage service implemented by the node and, in particular, the storageI/O stack 300. A suitable operating system kernel 224 may include ageneral-purpose operating system, such as the UNIX® series or MicrosoftWindows® series of operating systems, or an operating system withconfigurable functionality such as microkernels and embedded kernels.However, in an embodiment described herein, the operating system kernelis illustratively the Linux® operating system. It will be apparent tothose skilled in the art that other processing and memory means,including various computer readable media, may be used to store andexecute program instructions pertaining to the embodiments herein.

Each storage controller 240 cooperates with the storage I/O stack 300executing on the node 200 to access information requested by the host120. The information is preferably stored on storage devices such assolid state drives (SSDs) 260, illustratively embodied as flash storagedevices, of storage array 150. In an embodiment, the flash storagedevices may be based on NAND flash components, e.g., single-layer-cell(SLC) flash, multi-layer-cell (MLC) flash or triple-layer-cell (TLC)flash, although it will be understood to those skilled in the art thatother non-volatile, solid-state electronic devices (e.g., drives basedon storage class memory components) may be advantageously used with theembodiments described herein. Accordingly, the storage devices may ormay not be block-oriented (i.e., accessed as blocks). The storagecontroller 240 includes one or more ports having I/O interface circuitrythat couples to the SSDs 260 over the storage interconnect 140,illustratively embodied as a serial attached SCSI (SAS) topology.Alternatively, other point-to-point I/O interconnect arrangements may beused, such as a conventional serial ATA (SATA) topology or a PCItopology. The system interconnect 270 may also couple the node 200 to alocal service storage device 248, such as an SSD, configured to locallystore cluster-related configuration information, e.g., as clusterdatabase (DB) 244, which may be replicated to the other nodes 200 in thecluster 100.

The cluster interconnect interface 250 may include one or more portsadapted to couple the node 200 to the other node(s) of the cluster 100.In an embodiment, Ethernet may be used as the clustering protocol andinterconnect fabric media, although it will be apparent to those skilledin the art that other types of protocols and interconnects, such asInfiniBand, may be utilized within the embodiments described herein. TheNVRAM 280 may include a back-up battery or other built-in last-stateretention capability (e.g., non-volatile semiconductor memory such asstorage class memory) that is capable of maintaining data in light of afailure to the node and cluster environment. Illustratively, a portionof the NVRAM 280 may be configured as one or more non-volatile logs(NVLogs 285) configured to temporarily record (“log”) I/O requests, suchas write requests, received from the host 120.

Storage I/O Stack

FIG. 3 is a block diagram of the storage I/O stack 300 that may beadvantageously used with one or more embodiments described herein. Thestorage I/O stack 300 includes a plurality of software modules or layersthat cooperate with other functional components of the nodes 200 toprovide the distributed storage architecture of the cluster 100. In anembodiment, the distributed storage architecture presents an abstractionof a single storage container, i.e., all of the storage arrays 150 ofthe nodes 200 for the entire cluster 100 organized as one large pool ofstorage. In other words, the architecture consolidates storage, i.e.,the SSDs 260 of the arrays 150, throughout the cluster (retrievable viacluster-wide keys) to enable storage of the LUNs. Both storage capacityand performance may then be subsequently scaled by adding nodes 200 tothe cluster 100.

Illustratively, the storage I/O stack 300 includes an administrationlayer 310, a protocol layer 320, a persistence layer 330, a volume layer340, an extent store layer 350, a RAID layer 360, a storage layer 365and a NVRAM (storing NVLogs) “layer” interconnected with a messagingkernel 370. The messaging kernel 370 may provide a message-based (orevent-based) scheduling model (e.g., asynchronous scheduling) thatemploys messages as fundamental units of work exchanged (i.e., passed)among the layers. Suitable message-passing mechanisms provided by themessaging kernel to transfer information between the layers of thestorage I/O stack 300 may include, e.g., for intra-node communication:i) messages that execute on a pool of threads, ii) messages that executeon a single thread progressing as an operation through the storage I/Ostack, iii) messages using an Inter Process Communication (IPC)mechanism, and, e.g., for inter-node communication: messages using aRemote Procedure Call (RPC) mechanism in accordance with a functionshipping implementation. Alternatively, the I/O stack may be implementedusing a thread-based or stack-based execution model. In one or moreembodiments, the messaging kernel 370 allocates processing resourcesfrom the operating system kernel 224 to execute the messages. Eachstorage I/O stack layer may be implemented as one or more instances(i.e., processes) executing one or more threads (e.g., in kernel or userspace) that process the messages passed between the layers such that themessages provide synchronization for blocking and non-blocking operationof the layers.

In an embodiment, the protocol layer 320 may communicate with the host120 over the network 130 by exchanging discrete frames or packetsconfigured as I/O requests according to pre-defined protocols, such asiSCSI and FCP. An I/O request, e.g., a read or write request, may bedirected to a LUN and may include I/O parameters such as, inter alia, aLUN identifier (ID), a logical block address (LBA) of the LUN, a length(i.e., amount of data) and, in the case of a write request, write data.The protocol layer 320 receives the I/O request and forwards it to thepersistence layer 330, which records the request into a persistentwrite-back cache 380 illustratively embodied as a log whose contents canbe replaced randomly, e.g., under some random access replacement policyrather than only in serial fashion, and returns an acknowledgement tothe host 120 via the protocol layer 320. In an embodiment only I/Orequests that modify the LUN, e.g., write requests, are logged. Notably,the I/O request may be logged at the node receiving the I/O request, orin an alternative embodiment in accordance with the function shippingimplementation, the I/O request may be logged at another node.

Illustratively, dedicated logs may be maintained by the various layersof the storage I/O stack 300. For example, a dedicated log 335 may bemaintained by the persistence layer 330 to record the I/O parameters ofan I/O request as equivalent internal, i.e., storage I/O stack,parameters, e.g., volume ID, offset, and length. In the case of a writerequest, the persistence layer 330 may also cooperate with the NVRAM 280to implement the write-back cache 380 configured to store the write dataassociated with the write request. In an embodiment, the write-backcache may be structured as a log. Notably, the write data for the writerequest may be physically stored in the cache 380 such that the log 355contains the reference to the associated write data. It will beunderstood to persons skilled in the art that other variations of datastructures may be used to store or maintain the write data in NVRAMincluding data structures with no logs. In an embodiment, a copy of thewrite-back cache may be also maintained in the memory 220 to facilitatedirect memory access to the storage controllers. In other embodiments,caching may be performed at the host 120 or at a receiving node inaccordance with a protocol that maintains coherency between the datastored at the cache and the cluster.

In an embodiment, the administration layer 310 may apportion the LUNinto multiple volumes, each of which may be partitioned into multipleregions (e.g., allotted as disjoint block address ranges), with eachregion having one or more segments stored as multiple stripes on thearray 150. A plurality of volumes distributed among the nodes 200 maythus service a single LUN, i.e., each volume within the LUN services adifferent LBA range (i.e., offset and length, hereinafter offset range)or set of ranges within the LUN. Accordingly, the protocol layer 320 mayimplement a volume mapping technique to identify a volume to which theI/O request is directed (i.e., the volume servicing the offset rangeindicated by the parameters of the I/O request). Illustratively, thecluster database 244 may be configured to maintain one or moreassociations (e.g., key-value pairs) for each of the multiple volumes,e.g., an association between the LUN ID and a volume, as well as anassociation between the volume and a node ID for a node managing thevolume. The administration layer 310 may also cooperate with thedatabase 244 to create (or delete) one or more volumes associated withthe LUN (e.g., creating a volume ID/LUN key-value pair in the database244). Using the LUN ID and LBA (or LBA range), the volume mappingtechnique may provide a volume ID (e.g., using appropriate associationsin the cluster database 244) that identifies the volume and nodeservicing the volume destined for the request as well as translate theLBA (or LBA range) into an offset and length within the volume.Specifically, the volume ID is used to determine a volume layer instancethat manages volume metadata associated with the LBA or LBA range. Asnoted, the protocol layer 320 may pass the I/O request (i.e., volume ID,offset and length) to the persistence layer 330, which may use thefunction shipping (e.g., inter-node) implementation to forward the I/Orequest to the appropriate volume layer instance executing on a node inthe cluster based on the volume ID.

In an embodiment, the volume layer 340 may manage the volume metadataby, e.g., maintaining states of host-visible containers, such as rangesof LUNs, and performing data management functions, such as creation ofsnapshots and clones, for the LUNs in cooperation with theadministration layer 310. The volume metadata is illustratively embodiedas in-core mappings from LUN addresses (i.e., offsets) to durable extentkeys, which are unique cluster-wide IDs associated with SSD storagelocations for extents within an extent key space of the cluster-widestorage container. That is, an extent key may be used to retrieve thedata of the extent at an SSD storage location associated with the extentkey. Alternatively, there may be multiple storage containers in thecluster wherein each container has its own extent key space, e.g., wherethe administration layer 310 provides distribution of extents among thestorage containers. An extent is a variable length block of data thatprovides a unit of storage on the SSDs and that need not be aligned onany specific boundary, i.e., it may be byte aligned. Accordingly, anextent may be an aggregation of write data from a plurality of writerequests to maintain such alignment. Illustratively, the volume layer340 may record the forwarded request (e.g., information or parameterscharacterizing the request), as well as changes to the volume metadata,in dedicated log 345 maintained by the volume layer 340. Subsequently,the contents of the volume layer log 345 may be written to the storagearray 150 in accordance with a checkpoint (e.g., synchronization)operation that stores in-core metadata on the array 150. That is, thecheckpoint operation (checkpoint) ensures that a consistent state ofmetadata, as processed in-core, is committed to (i.e., stored on) thestorage array 150; whereas the retirement of log entries ensures thatthe entries accumulated in the volume layer log 345 synchronize with themetadata checkpoints committed to the storage array 150 by, e.g.,retiring those accumulated log entries that are prior to the checkpoint.In one or more embodiments, the checkpoint and retirement of log entriesmay be data driven, periodic or both.

In an embodiment, the extent store layer 350 is responsible for storingextents on the SSDs 260 (i.e., on the storage array 150) and forproviding the extent keys to the volume layer 340 (e.g., in response toa forwarded write request). The extent store layer 350 is alsoresponsible for retrieving data (e.g., an existing extent) using anextent key (e.g., in response to a forwarded read request). The extentstore layer 350 may be responsible for performing de-duplication andcompression on the extents prior to storage. The extent store layer 350may maintain in-core mappings (e.g., embodied as hash tables) of extentkeys to SSD storage locations (e.g., offset on an SSD 260 of array 150).The extent store layer 350 may also maintain a dedicated log 355 ofentries that accumulate requested “put” and “delete” operations (i.e.,write requests and delete requests for extents issued from other layersto the extent store layer 350), where these operations change thein-core mappings (i.e., hash table entries). Subsequently, the in-coremappings and contents of the extent store layer log 355 may be writtento the storage array 150 in accordance with a “fuzzy” checkpoint 390(i.e., checkpoints with incremental changes recorded in the one or morelog files) in which selected in-core mappings (less than the total) arecommitted to the array 150 at various intervals (e.g., driven by anamount of change to the in-core mappings, size thresholds of log 355, orperiodically). Notably, the accumulated entries in log 355 may beretired once all in-core mappings have been committed to include changesrecorded in those entries.

In an embodiment, the RAID layer 360 may organize the SSDs 260 withinthe storage array 150 as one or more RAID groups (e.g., sets of SSDs)that enhance the reliability and integrity of extent storage on thearray by writing data “stripes” having redundant information, i.e.,appropriate parity information with respect to the striped data, acrossa given number of SSDs 260 of each RAID group. The RAID layer 360 mayalso store a number of stripes (e.g., stripes of sufficient depth),e.g., in accordance with a plurality of contiguous range writeoperations, so as to reduce data relocation (i.e., internal flash blockmanagement) that may occur within the SSDs as a result of theoperations. In an embodiment, the storage layer 365 implements storageI/O drivers that may communicate directly with hardware (e.g., thestorage controllers and cluster interface) cooperating with theoperating system kernel 224, such as a Linux virtual function I/O (VFIO)driver.

Write Path

FIG. 4 illustrates an I/O (e.g., write) path 400 of the storage I/Ostack 300 for processing an I/O request, e.g., a SCSI write request 410.The write request 410 may be issued by host 120 and directed to a LUNstored on the storage arrays 150 of the cluster 100. Illustratively, theprotocol layer 320 receives and processes the write request by decoding420 (e.g., parsing and extracting) fields of the request, e.g., LUN ID,LBA and length (shown at 413), as well as write data 414. The protocollayer 320 may use the results 422 from decoding 420 for a volume mappingtechnique 430 (described above) that translates the LUN ID and LBA range(i.e., equivalent offset and length) of the write request to anappropriate volume layer instance, i.e., volume ID (volume 445), in thecluster 100 that is responsible for managing volume metadata for the LBArange. In an alternative embodiment, the persistence layer 330 mayimplement the above described volume mapping technique 430. The protocollayer then passes the results 432, e.g., volume ID, offset, length (aswell as write data), to the persistence layer 330, which records therequest in the persistence layer log 335 and returns an acknowledgementto the host 120 via the protocol layer 320. The persistence layer 330may aggregate and organize write data 414 from one or more writerequests into a new extent 610 and perform a hash computation, i.e., ahash function, on the new extent to generate a hash value 650 inaccordance with an extent hashing technique 450.

The persistence layer 330 may then pass the write request withaggregated write data including, e.g., the volume ID, offset and length,as parameters 434 to the appropriate volume layer instance. In anembodiment, message passing of the parameters 434 (received by thepersistence layer) may be redirected to another node via the functionshipping mechanism, e.g., RPC, for inter-node communication.Alternatively, message passing of the parameters 434 may be via the IPCmechanism, e.g., message threads, for intra-node communication.

In one or more embodiments, a bucket mapping technique 670 is providedthat translates the hash value 650 to an instance 470 of an appropriateextent store layer (i.e., extent store instance 470) that is responsiblefor storing the new extent 610. The bucket mapping technique may beimplemented in any layer of the storage I/O stack above the extent storelayer. In an embodiment, for example, the bucket mapping technique maybe implemented in the persistence layer 330, the volume layer 340, or alayer that manages cluster-wide information, such as a cluster layer(not shown). Accordingly, the persistence layer 330, the volume layer340, or the cluster layer may contain computer executable instructionsexecuted by the CPU 210 to perform operations that implement the bucketmapping technique 670 described herein. The persistence layer 330 maythen pass the hash value 650 and the new extent 610 to the appropriatevolume layer instance and onto the appropriate extent store instance viaan extent store put operation. The extent hashing technique 450 mayembody an approximately uniform hash function to ensure that any randomextent to be written may have an approximately equal chance of fallinginto any extent store instance 470, i.e., hash buckets are distributedacross extent store instances of the cluster 100 based on availableresources. As a result, the bucket mapping technique 670 providesload-balancing of write operations (and, by symmetry, read operations)across nodes 200 of the cluster, while also leveling flash wear in theSSDs 260 of the cluster.

In response to the put operation, the extent store instance may processthe hash value 650 to perform an extent metadata selection technique 460that (i) selects an appropriate hash table 480 (e.g., hash table 480 a)from a set of hash tables (illustratively in-core) within the extentstore instance 470, and (ii) extracts a hash table index 462 from thehash value 650 to index into the selected hash table and lookup a tableentry having an extent key 475 identifying a storage location 490 on SSD260 for the extent. Accordingly, the extent store layer 350 containscomputer executable instructions executed by the CPU 210 to performoperations that implement the metadata selection technique 460 describedherein. If a table entry with a matching key is found, then SSD location490 mapped from the extent key 475 is used to retrieve an existingextent (not shown) from SSD. The existing extent is then compared withthe new extent 610 to determine whether their data is identical. If thedata is identical, the new extent 610 is already stored on SSD 260 and ade-duplication opportunity (denoted de-duplication 452) exists such thatthere is no need to write another copy of the data. Accordingly, areference count in the table entry for the existing extent isincremented and the extent key 475 of the existing extent is passed tothe appropriate volume layer instance for storage within an entry(denoted as volume metadata entry 446) of a dense tree metadatastructure 444 (e.g., dense tree 444 a), such that the extent key 475 isassociated an offset range 440 (e.g., offset range 440 a) of the volume445.

However, if the data of the existing extent is not identical to the dataof the new extent 610, a collision occurs and a deterministic algorithmis invoked to sequentially generate as many new candidate extent keys(not shown) mapping to the same bucket as needed to either providede-duplication 452 or produce an extent key that is not already storedwithin the extent store instance. Notably, another hash table (e.g.,hash table 480 n) of extent store instance 470 may be selected by a newcandidate extent key in accordance with the extent metadata selectiontechnique 460. In the event that no de-duplication opportunity exists(i.e., the extent is not already stored) the new extent 610 iscompressed in accordance with compression technique 454 and passed tothe RAID layer 360, which processes the new extent 610 for storage onSSD 260 as a stripe 810 of RAID group 466. The extent store instance maycooperate with the RAID layer 360 to identify a storage segment 710(i.e., a portion of the storage array 150) and a location on SSD 260within the segment 710 in which to store the new extent 610.Illustratively, the identified storage segment is a segment with a largecontiguous free space having, e.g., location 490 on SSD 260 b forstoring the extent 610.

In an embodiment, the RAID layer 360 then writes the stripe 810 acrossthe RAID group 466, illustratively as a full stripe write 458. The RAIDlayer 360 may write a series of stripes 810 of sufficient depth toreduce data relocation that may occur within flash-based SSDs 260 (i.e.,flash block management). The extent store instance then (i) loads theSSD location 490 of the new extent 610 into the selected hash table 480n (i.e., as selected by the new candidate extent key), (ii) passes a newextent key (denoted as extent key 475) to the appropriate volume layerinstance for storage within an entry (also denoted as volume metadataentry 446) of a dense tree 444 managed by that volume layer instance,and (iii) records a change to extent metadata of the selected hash tablein the extent store layer log 355. Illustratively, the volume layerinstance selects dense tree 444 a spanning an offset range 440 a of thevolume 445 that encompasses the offset range of the write request. Asnoted, the volume 445 (e.g., an offset space of the volume) ispartitioned into multiple regions (e.g., allotted as disjoint offsetranges); in an embodiment, each region is represented by a dense tree444. The volume layer instance then inserts the volume metadata entry446 into the dense tree 444 a and records a change corresponding to thevolume metadata entry in the volume layer 345. Accordingly, the I/O(write) request is sufficiently stored on SSD 260 of the cluster.

Read Path

FIG. 5 illustrates an I/O (e.g., read) path 500 of the storage I/O stack300 for processing an I/O request, e.g., a SCSI read request 510. Theread request 510 may be issued by host 120 and received at the protocollayer 320 of a node 200 in the cluster 100. Illustratively, the protocollayer 320 processes the read request by decoding 420 (e.g., parsing andextracting) fields of the request, e.g., LUN ID, LBA, and length (shownat 513), and uses the decoded results 522, e.g., LUN ID, offset, andlength, for the volume mapping technique 430. That is, the protocollayer 320 may implement the volume mapping technique 430 (describedabove) to translate the LUN ID and LBA range (i.e., equivalent offsetand length) of the read request to an appropriate volume layer instance,i.e., volume ID (volume 445), in the cluster 100 that is responsible formanaging volume metadata for the LBA (i.e., offset) range. The protocollayer then passes the results 532 to the persistence layer 330, whichmay search the write-back cache 380 to determine whether some or all ofthe read request can be serviced from its cached data. If the entirerequest cannot be serviced from the cached data, the persistence layer330 may then pass the read request including, e.g., the volume ID,offset and length, as parameters 534 to the appropriate volume layerinstance in accordance with the function shipping mechanism, e.g., RPC,for inter-node communication or the IPC mechanism, e.g., messagethreads, for intra-node communication.

The volume layer instance may process the read request to access a densetree metadata structure 444 (e.g., dense tree 444 a) associated with aregion (e.g., offset range 440 a) of a volume 445 that encompasses therequested offset range (specified by parameters 534). The volume layerinstance may further process the read request to search for (lookup) oneor more volume metadata entries 446 of the dense tree 444 a to obtainone or more extent keys 475 associated with one or more extents 610 (orportions of extents) within the requested offset range. In anembodiment, each dense tree 444 may be embodied as multiple levels of asearch structure with possibly overlapping offset range entries at eachlevel. The various levels of the dense tree may have volume metadataentries 446 for the same offset, in which case, the higher level has thenewer entry and is used to service the read request. A top level of thedense tree 444 is illustratively resident in-core and a page cache 448may be used to access lower levels of the tree. If the requested rangeor portion thereof is not present in the top level, a metadata pageassociated with an index entry at the next lower tree level (not shown)is accessed. The metadata page (i.e., in the page cache 448) at the nextlevel is then searched (e.g., a binary search) to find any overlappingentries. This process is then iterated until one or more volume metadataentries 446 of a level are found to ensure that the extent key(s) 475for the entire requested read range are found. If no metadata entriesexist for the entire or portions of the requested read range, then themissing portion(s) are zero filled.

Once found, each extent key 475 is processed by the volume layer 340 to,e.g., implement the bucket mapping technique 670 that translates theextent key to an appropriate extent store instance 470 responsible forstoring the requested extent 610. Note that, in an embodiment, eachextent key 475 may be substantially identical to the hash value 650associated with the extent 610, i.e., the hash value as calculatedduring the write request for the extent, such that the bucket mapping670 and extent metadata selection 460 techniques may be used for bothwrite and read path operations. Note also that the extent key 475 may bederived from the hash value 650. The volume layer 340 may then pass theextent key 475 (i.e., the hash value from a previous write request forthe extent) to the appropriate extent store instance 470 (via an extentstore get operation), which performs an extent key-to-SSD mapping todetermine the location on SSD 260 for the extent.

In response to the get operation, the extent store instance may processthe extent key 475 (i.e., hash value 650) to perform the extent metadataselection technique 460 that (i) selects an appropriate hash table 480(e.g., hash table 480 a) from a set of hash tables within the extentstore instance 470, and (ii) extracts a hash table index 462 from theextent key 475 (i.e., hash value 650) to index into the selected hashtable and lookup a table entry having a matching extent key 475 thatidentifies a storage location 490 on SSD 260 for the extent 610. Thatis, the SSD location 490 mapped to the extent key 475 may be used toretrieve the existing extent (denoted as extent 610) from SSD 260 (e.g.,SSD 260 b). The extent store instance then cooperates with the RAIDlayer 360 to access the extent on SSD 260 b and retrieve the datacontents in accordance with the read request. Illustratively, the RAIDlayer 360 may read the extent in accordance with an extent readoperation 468 and pass the extent 610 to the extent store instance. Theextent store instance may then decompress the extent 610 in accordancewith a decompression technique 456, although it will be understood tothose skilled in the art that decompression can be performed at anylayer of the storage I/O stack 300. The extent 610 may be stored in abuffer (not shown) in memory 220 and a reference to that buffer may bepassed back through the layers of the storage I/O stack 300. Thepersistence layer may then load the extent into a read cache 580 (orother staging mechanism) and may extract appropriate read data 512 fromthe read cache 580 for the LBA range of the read request 510.Thereafter, the protocol layer 320 may create a SCSI read response 514,including the read data 512, and return the read response to the host120.

Log-Structured Layer of File System

The embodiments described herein are directed to a flash-optimized,log-structured layer of a file system of the storage I/O stack. Thelog-structured layer (i.e., extent store layer) of the file systemprovides sequential storage of data and metadata (i.e., log-structuredlayout) on the SSDs 260 of the cluster to reduce write amplification,while leveraging the variable compression and variable length extentfeatures, as well as the extent de-duplication feature, of the storageI/O stack 300. The data may be organized as an arbitrary number ofvariable-length extents of one or more host-visible LUNs served by thenodes and stored as extents. The metadata may include mappings fromhost-visible logical block address ranges (i.e., offset ranges) of a LUNto extent keys (e.g., volume layer metadata), as well as mappings of theextent keys to SSD storage locations of the extents (e.g., extent storelayer metadata). Illustratively, the volume layer cooperates with theextent store layer to provide a level of indirection that facilitatesefficient log-structured layout of extents on the SSDs by the extentstore layer. That is, the extent key mappings maintained by the volumelayer allow relocation of the extents on SSD during, e.g., segmentcleaning, without update to the volume layer mappings. Accordingly, thestorage location of an extent on SSD is effectively “virtualized” by itsmapped extent key (i.e., extent store layer mappings) such thatrelocation of the extent on SSD does not require update to volume layermetadata (i.e., the extent key sufficiently identifies the extent). Thevirtualization of the storage locations also permits cleaning processesto occur in the extent store layer without update to volume layermetadata, thereby substantially reducing write amplification.

In an embodiment, the mappings of the extent keys to SSD storagelocations are performed and maintained by the extent store layer, whilethe mappings of the LUN offset ranges to the extent keys are performedand maintained by the volume layer. Separation of these mappingfunctions between the volume and extent store layers enables differentvolumes with different offset ranges to reference (map to) a same extentkey (and thus a same extent). Notably, separation of the volume layerand extent store layer of the layered file system enables efficientperformance of inline de-duplication that illustratively ensures thatthere is only one copy of each extent stored on the storage arrays ofthe cluster. Such assurance is global to the cluster as the single copyof the stored extent may span volumes and nodes of the cluster. Notably,de-duplication may be selectively applied only to data and not metadata(e.g., volume and extent store layer mappings), so as to reduce latencyof metadata operations, e.g., writes of metadata. In an embodiment,selective de-duplication may be accomplished by passing a flag in awrite operation to the extent store layer.

FIG. 6 illustrates a layered file system that may be advantageously usedwith one or more embodiments described herein. A plurality of writerequests 410 a,b, each directed to a different LUN having identicalwrite data 414, may be received by a node 200 a of the cluster 100. Anidentical hash value 650 a computed from the write data of each writerequest 410 a,b may lead to inline de-duplication (i.e., de-duplicationbefore storage on SSD, as described previously) of that data within anextent store 620 a (i.e., bucket). The dense trees 444 a,b for eachrespective LUN (representing LUN offset ranges 440 a,b respectively) mayreference the same extent 610 a (i.e., may store the same extent key475, not shown). Similarly, another plurality of write requests 410 c,dhaving different identical write data 415 received by a node 200 b ofthe cluster may lead to de-duplication of that data in another extentstore 620 b. Thus, the bucket mapping 670 of the hash value 650 may leadto a different extent store 620 b for data 415 than for data 414 (whichmay lead to extent store 620 a). Similarly, the dense trees 444 a,b foreach respective LUN (representing LUN offset ranges 440 a,brespectively) may reference the same extent 610 b (i.e., may store thesame extent key 475, not shown).

As noted, the persistence layer 330 may compute a hash value 650 on anextent 610 to determine which extent store instance 470 (or extentstore) is associated with the extent in accordance with the bucketmapping technique 670. The persistence layer may then pass the hashvalue 650 to the appropriate volume layer instance, which then passes onthe hash value to the appropriate extent store instance via an extentstore put operation. The extent store instance may determine whether theextent is previously stored on SSD in accordance with a de-duplicationopportunity. If the extent is not stored on the storage arrays of thecluster (i.e., anywhere in the cluster), the extent store instance mayform a unique extent key 475 from the hash value 650 prior to storingthe extent (as compressed) on SSD and return that unique key to thevolume layer instance. However, if it is determined that the extent isstored on any of the storage arrays in the cluster, the extent storeinstance may return the extent key for that stored extent to the volumelayer instance, thereby enabling global inline de-duplication (i.e.,de-duplication before storage on SSD) that obviates the need for aduplicate copy of the extent. Thus, the inline global de-duplicationopportunity arises from (and thus provides a motivation for) theseparation of the file system functions among the layers. Notably, thevolume layer is unaware of de-duplicated data stored only once in theunderlying extent store layer. Facilitation of bucket mapping via a hashspace and the resulting distribution of data and metadata among theextent store instances of the cluster also arise from the separation ofthe file system functions among the layers. That is, the volume layer isalso unaware of which extent store instance stores an extent, as extentkeys are global within the cluster. Thus, the benefit of inline globalde-duplication of data and distribution of data (and metadata) withinthe cluster both result from a separation of the file system functionsamong the layers.

Advantageously, the separation of the volume and extent store layerspermits a storage location of an extent on SSD to be effectivelyvirtualized by its mapped extent key such that relocation of the extenton SSD does not require update to volume layer metadata (i.e., theextent key sufficiently identifies the extent). As noted, virtualizationof the storage locations also permits cleaning processes to occur in theextent store layer without update to volume layer metadata, therebysubstantially reducing write amplification.

In an embodiment, functions of the log-structured layer of the filesystem, such as write allocation and flash device (i.e., SSD)management, are performed and maintained by the extent store layer.Write allocation may include gathering of the variable-length extents toform full stripes that may be written to free segments across SSDs ofone or more RAID groups. That is, the log-structured layer of the filesystem writes extents to initially free (i.e., clean) segments as fullstripes rather than partial stripes. Flash device management may includesegment cleaning to create such free segments that indirectly map toSSDs via the RAID groups. Accordingly, partial RAID stripe writes areavoided, which results in reduced RAID-related write amplification.

In addition, instead of relying on garbage collection in the SSDs, thestorage I/O stack may implement segment cleaning (i.e., garbagecollection) in the extent store layer to bypass performance impacts offlash translation layer (FTL) functionality (including garbagecollection) in the SSD. In other words, the storage I/O stack allows thelog-structured layer of the file system to operate as a data layoutengine using segment cleaning to effectively replace a substantialportion of the FTL functionality of the SSD. The extent store layer maythus process random write requests in accordance with segment cleaning(i.e., garbage collection) to predict flash behavior within its FTLfunctionality. As a result, a log-structured equivalent source of writeamplification for the storage I/O stack may be consolidated, andmanaged, at the extent store layer so that it subsumes (i.e., proxies)RAID-related and SSD related (i.e., FTL functionality) writeamplification. That is, the log-structured nature of the extent storelayer may be used to control and, thus, reduce both RAID-related writeamplification and SSD-related write amplification. Further, thelog-structured layer of the file system described herein may beemployed, in part, to improve write performance from the flash devicesof the storage array.

As noted, the log-structured layout of SSDs is realized by sequentiallywriting extents to clean segments. Thus, the log-structured layout(i.e., sequential storage) employed by the extent store layer inherentlysupports variable length extents, thereby allowing unrestrictedcompression of extents prior to storage on SSD and without specificblock level (i.e., in SSD blocks) metadata support from the SSDs, suchas 520 byte sectors supporting 512 bytes of data and 8 bytes of metadata(e.g., a pointer to another block containing a tail-end of compresseddata.) Typically, consumer grade SSDs support sectors as powers of 2(e.g., 512 bytes); whereas more expensive enterprise grade SSDs maysupport enhanced sized sectors (e.g., 520 bytes). Accordingly, theextent store layer may operate with lower cost consumer grade SSDs whilesupporting variable length extents with their concomitant unfetteredcompression.

Segment Cleaning

FIG. 7a illustrates segment cleaning by the log-structured layer of thefile system. In an embodiment, the extent store layer 350 of the layeredfile system may write extents to an empty or free region or “segment”.Before rewriting that segment again, the extent store layer 350 mayclean the segment in accordance with segment cleaning which,illustratively, may be embodied as a segment cleaning process. That is,the segment cleaning process may read all valid extents 610 from an oldsegment 710 a and write those valid extents (i.e., extents not deletedor overwritten 611) to one or more new segments 710 b,c, to therebyfree-up (i.e., “clean”) the old segment 710 a. New extents may then bewritten sequentially to the old (now clean) segment. The log-structuredlayer may maintain a certain amount of reserve space (i.e., freesegments) to enable efficient performance of segment cleaning. Forexample, the log-structured layer (i.e., extent store layer) mayillustratively maintain a reserve space of free segments equivalent toapproximately 7% of storage capacity. The sequential writing of newextents may manifest as full stripe writes 458, such that a singlestripe write operation to storage spans all SSDs in a RAID group 466.That is, write data may be accumulated until a stripe write operation ofa minimum depth (e.g., 64 KB) can be made.

Illustratively, segment cleaning may be performed to free one or moreselected segments that indirectly map to SSDs. As used herein, a SSD maybe composed of a plurality of segment chunks 720, wherein each chunk isillustratively approximately 1 GB in size. A segment may include onesegment chunk 720 a-c from each of a plurality of SSDs in a RAID group466. Thus, for a RAID group having 24 SSDs, wherein the equivalentstorage space of 22 SSDs store data (data SSDs) and the equivalentstorage space of 2 SSDs store parity (parity SSDs), each segment mayinclude 22 GB of data and 2 GB of parity. The RAID layer may furtherconfigure the RAID groups according to one or more RAID implementations,e.g., RAID 1, 4, 5 and/or 6, to thereby provide protection over the SSDsin the event of, e.g., failure to one or more SSDs. Notably, eachsegment may be associated with a different RAID group and, thus, mayhave a different RAID configuration, i.e., each RAID group may beconfigured according to a different RAID implementation. To free-up orclean selected segments, extents of the segments that contain valid dataare moved to different clean segments and the selected segments (nowclean) are freed for subsequent reuse. Segment cleaning consolidatesfragmented free space to improve write efficiency, e.g., to stripes byreducing RAID-related write amplification and to underlying flash blocksby reducing performance impacts of the FTL. Once a segment is cleanedand designated freed, data may be written sequentially to that segment.Accounting structures, e.g., free segment maps or an amount of segmentfree space, maintained by the extent store layer for write allocation,may be employed by the segment cleaning process. Notably, selection of aclean segment to receive data (i.e., writes) from a segment beingcleaned may be based upon the amount of free space remaining in theclean segment and/or the last time the clean segment was used. Notefurther that different portions of data from the segment being cleanedmay be moved to different “target” segments. That is, a plurality ofrelatively clean segments 710 b,c may receive differing portions of datafrom the segment 710 a being cleaned.

In an embodiment, segment cleaning may cause some write amplification inthe storage array (SSDs). However, the file system may reduce such writeamplification by writing extents to the SSDs sequentially as a logdevice. For example, given SSDs with an erase block size ofapproximately 2 MBs, by writing at least 2 MB of data (extents)sequentially to a free segment, an entire erase block may be overwrittenand fragmentation at the SSD level may be eliminated (i.e., reducinggarbage collection in the SSD). However, the SSDs typically stripe dataacross multiple flash components and across multiple channels in orderto realize performance. Thus, a relatively large (e.g., 1 GB) writegranularity to a free (i.e., clean) segment may be necessary to avoidwrite amplification at the SSD level (i.e., to override internal SSDstriping).

Specifically because the erase block boundaries in the SSD may beunknown, the write granularity should be large enough so that a sequenceof writes for extents over a large contiguous range may overwritepreviously written extents on the SSD and effectively override garbagecollection in the SSDs. In other words, such garbage collection may bepreempted because the new data is written over the same range asprevious data such that the new data completely overwrites thepreviously written data. This approach also reduces consumption of thereserve space capacity with the new write data. Accordingly, anadvantage of the log-structured feature of the storage I/O stack (i.e.,log-structured layer in the file system) is the ability to reduce writeamplification of the SSDs with only a minimum amount of reserve space inthe SSDs. This log-structured feature effectively “moves” flash devicemanagement of reserve space from the SSD to the extent store layer,which uses that reserve space to manage the write amplification. Thus,instead of having two sources of write amplification (i.e., RAID-relatedoperations and the SSD FTL, which multiply) there is only one source ofwrite amplification (i.e., the extent store layer). That is, thelog-structured nature of the extent store layer may be used to controland, thus, reduce both RAID-related write amplification and SSD-relatedwrite amplification.

Segment Policies

As noted above, a plurality of segments may be involved during segmentcleaning. In an embodiment, the log-structured layer of the file systemmay employ data structures to maintain information for heuristics andpolicies directed to, e.g., classification of segments based on data aswell as metadata describing the layout of the data on SSDs of thestorage array. Assume, for example, that the layered file systemservices a random write operation workload and, after extent hashing,initially stores the associated write data as one or more “hot” extentson the SSDs, which extents subsequently become “cold.” Illustratively,the classification of an extent as “cold” or “hot” may be determined bya policy that models (i.e., predicts) the expected longevity (i.e.,before deletion) of the extent based on factors, such as when the extentwas last accessed or a type (e.g., video, text) of data in the extent.Such a policy may further facilitate separation of hot and cold extentsbased on, e.g., ages of the extents. For example, if an extent hassurvived N segment cleanings and, therefore, has been copied N times bythe segment cleaning process, a decision may be rendered that the extentmay continue to be retained for a long period of time and thus is“cold.” Accordingly, the extent may be stored in a cold segment ratherthan a hot segment. The algorithm used to determine the age at which anextent may be declared “hot” or “cold” may vary based on thedistribution of ages of the extents within an extent store instance. Asa result, hot segments (i.e., hot extents moved into them) may requiremore frequent cleaning than cold segments (i.e., cold extents moved intothem), which change less frequently. The classification of a segment(and its extents) can be further extended to various gradients of hotand cold, e.g., “very hot,” “hot,” “cold,” “very cold”. Suchclassifications may represent discrete points along a continuousspectrum of expected extent longevity.

As used herein, a log-structured layout capability may denote writing ofthe data (or metadata) in a pattern that is efficient forsequentially-accessed devices. More specifically, a log-structuredtechnique may convert write data associated with write operations fromthe host that have “temporal locality” (i.e., are performed closetogether in time) to a layout that has “spatial locality” on the SSD,even though the data may not be spatially local in the address space asviewed by the host. That is, the host may consider the data to be random(i.e., written at random times), but because the data is either receivedclose in time (e.g., a “burst” of writes) at a node or is de-stagedclose in time by the node using, e.g., the write-back cache, the data iswritten to the persistent storage proximately (i.e., with spatiallocality) on the device (i.e., SSD) in order to get better writeperformance out of the device. By employing log-structured capability,the file system may flush (write) data associated with unrelated(random) write operations to SSD in a pattern that is efficient forextracting write performance from the SSD (i.e., the log-structuredcapability transposes the random write operations from the host tosequential write operations at the node for efficient storage on theSSD). In the case of flash storage devices, it may be advantageous thatthe data be written in a specific sequence or order (e.g., largecontiguous range) as a group to a segment (i.e., a segment write cycle)to reduce, e.g., performance impact of FTL in the SSD.

FIG. 7b illustrates hot and cold segments used by the log-structuredlayer (i.e., extent store layer) of the file system. In an embodiment,the log-structured layout applied to SSD storage has various attributes.For example, data and metadata (extents) are written to clean segmentsand full RAID stripes are written to minimize parity overhead.Eventually, hot extents 610 and cold extents 612 may be split from eachother into distinct segments (i.e., hot segments 710 b and cold segments710 c) over a number of segment cleaning cycles. Because writing anddeleting of the extents are based on host data access patterns, they maybe non-uniform which may be exploited by splitting the extents betweenhot and cold. For example, a large amount of data (or metadata) may bewritten and then left for a relatively long time without being deletedor overwritten. An overwrite that occurs at the extent store layerillustratively manifests as an extent 611 being deleted and replaced byanother extent, because their extent keys are different. From the pointof view of the extent store layer, the non-uniform write patternmanifests as different extents being valid (i.e., not overwritten ordeleted) for different lengths of time, and the longer an extent isvalid, the longer it is likely to stay valid. Conversely, the morerecently an extent was written, the more likely it is to be deletedbefore it significantly ages. That is, more recent data is more likelyto be modified (i.e., overwritten or deleted), whereas older data isless likely to be even accessed.

Thus, segregating the extents into hot and cold segments facilitatescreation of entire segments of extents that are densely packed withcurrently valid data as the segments are cleaned. Because valid extentsare deleted at a much lower rate than extents in other segments, validdata segments (i.e., having few extents overwritten or deleted) have alower fraction of free space relative to the other segments.Accordingly, unnecessary write amplification is reduced as segments withgreater data change (i.e., hot segments with extents frequently beingoverwritten or deleted) may maintain a greater amount of free space thansegments with less data change (i.e., cold segments with infrequentlyaccessed data). That is, segments may be selected for cleaning based onthe amount of their free space. Hot and cold segments may be treateddifferently because a lower fraction of free space may be maintained inthe cold segments to realize a higher fraction of free space, onaverage, in the hot segments. By keeping some of the segments full ofcold data and with low free space, the overall free space may be lessfragmented and concentrated into the hot segments.

As their extents are deleted, the hot segments may end up with a higheraverage free space than the overall free space average. Unnecessarycopying of valid data may be reduced during segment cleaning becausesegments are organized into hot and cold segments, which yields greaterefficiency of (i.e., reduces) write amplification. That is, the amountof data that has to be rewritten elsewhere (i.e., to a clean segment)when cleaning a segment may be lower because more of that segment wasalready freed by the time it is cleaned. Conversely, consider a simplecase with uniform random writes in which all of the segments are treatedequally as hot or cold, and the segment with the highest free space atany given time is chosen for cleaning. For this case, if the full amountof reserved space on the SSD is, e.g., 20%, then the segments mostrecently cleaned are 0% free and, due to a snowplow effect (i.e.,accumulating of data changes over time), the segments that are oldestand about to be cleaned are at (roughly) a 40% free level.

In an embodiment, the cold segments may be cleaned at a lower level offree space than hot segments. In effect, the cold segments are cleanedless frequently so that they collectively have less overall free space(i.e., contain more data) leaving a greater amount of overall free spacefor the hot segments. Illustratively, for each hot and cold segment, thesegment with most free space may be chosen for cleaning in order to getthe best efficiency, i.e., the lowest amount of write amplification. Thesegment with the most amount of free space may be chosen for cleaningbecause it has the least amount of valid data that has to be relocated,thus yielding the lowest amount of unnecessary write amplification.Accordingly, to continually keep their free space low, cold segments maybe chosen for cleaning even though they have, e.g., only 8% free space.Notably, a low free space threshold may be maintained as long as therate of cleaning cold segments is relatively low compared to the rate ofcleaning hot segments, i.e., the percentage of free space in the hotsegments is relatively high as compared to the cold segments.

Write Allocation

In an embodiment, there may be multiple RAID stripes per segment. Eachtime a segment is allocated, i.e., after cleaning the segment, thechunks of various SSDs within the segment may include a series of RAIDstripes each aligned by extent. That is, the extents 610 may be formedinto chunks 720 and written to SSD as RAID stripes that remain aligned(i.e., logically) by extent within each chunk of a segment. Note thatthe chunks may be at the same or different offsets within the SSDs. Theextent store layer may read the chunks sequentially for cleaningpurposes and relocate all the valid data to another segment. Thereafter,the chunks 720 of the cleaned segment may be freed and a decision may berendered as to how to constitute the next segment that uses the chunks.For example, if a SSD is removed from a RAID group, a portion (i.e., aset of chunks 720) of capacity may be omitted from the next segment(i.e., change in RAID stripe configuration) so as to constitute the RAIDgroup from a plurality of chunks 720 that is one chunk narrower, i.e.,makes the RAID width one less. Thus, by using segment cleaning, a RAIDgroup of the chunks 720 constituting the segments may be effectivelycreated each time a new segment is allocated, i.e., a RAID group iscreated dynamically from available SSDs when a new segment is allocated.There is generally no requirement to include all of the SSDs 260 in thestorage array 150 in the new segment. Alternatively, a chunk 720 from anewly introduced SSD can be added into a RAID group created when a newsegment 710 is allocated.

FIG. 8 illustrates a RAID stripe formed by the log-structured layer ofthe file system. As noted, write allocation may include gathering of thevariable-length extents to form one or more stripes across SSDs of oneor more RAID groups. In an embodiment, the RAID layer 360 may manageparity computations and topology information used for placement of theextents 610 on the SSDs 260 a-n of the RAID group 466. To that end, theRAID layer may cooperate with the extent store layer to organize theextents as stripes 810 within the RAID group. Illustratively, the extentstore layer may gather the extents 610 to form one or more full stripes810 that may be written to a free segment 710 d such that a singlestripe write operation 458 may span all SSDs in that RAID group. Theextent store layer may also cooperate with the RAID layer to pack eachstripe 810 as a full stripe of variable-length extents 610. Once thestripe is complete, the RAID layer may pass the full stripe 810 ofextents to the storage layer 365 of the storage I/O stack for storage onthe SSDs 260. By writing a full stripe (i.e., data and parity) to thefree segment, the extent store layer (i.e., log-structured layer of thefile system) avoids the cost of parity updates and spreads any requiredread operation load across the SSDs. Notably, the extents 610 pending awrite operation on an SSD 260 may be accumulated into a chunk 720 d,e,which is written as one or more temporally proximate write operations tothe SSD (e.g., as 1 Gbyte), thereby reducing the performance impact ofthe FTL in the SSD.

In an embodiment, an extent store may be viewed as a global pool ofextents stored on the storage arrays 150 of the cluster, where eachextent may be maintained within a RAID group 466 of an extent storeinstance. Assume one or more variable-length (i.e., small and/or large)extents are written to a segment 710 d. The extent store layer maygather the variable-length extents to form one or more stripes acrossthe SSDs is of the RAID group. Although each stripe may include multipleextents 610 and an extent 610 b could span more than one stripe 810 a,b,each extent is entirely stored on one SSD. In an embodiment, a stripemay have a depth of 16 KB and an extent may have a size of 4 KB, but theextent may thereafter be compressed down to 1 or 2 KB or smallerpermitting a larger extent to be packed which may exceed the stripedepth (i.e., the chunk 720 depth). Thus, a stripe may constitute onlypart of the extent, so the depth of the stripe 810 (i.e., the set ofchunks 720 d-f constituting the stripe) may be independent of theextent(s) written to any one SSD. Since the extent store layer may writethe extents as full stripes across one or more free segments of theSSDs, write amplification associated with processing information of thestripes may be reduced.

Operationally, the extent store layer may send the extents to the RAIDlayer, which attempts to pack the stripes as much as possible with thecompressed, variable length extents. In an embodiment, a minimum unit ofwrite operation in the storage I/O stack may constitute a stripe depthof 4 KB, or 8 sectors of 512 or 520 bytes depending on the underlyingsector size of the SSD. Once a full stripe is complete, the RAID layermay pass the extents to the storage layer, which may cooperate with thepersistence layer to store the full stripe of extents on the SSDs.Notably, the size of write operations may vary depending on the writeload on the system. Illustratively write operations may vary from aminimum of about 4 KB chunks per SSD under a light write load, to amaximum of 64 KB chunks per SSD under a heavy write load, which may bedetermined by pressure in the persistent write-back cache 380. As such,the persistence layer may wait until enough write data is accumulated inthe write-back cache (i.e., a larger write size) before writing data(i.e., larger chunk sizes) to SSD. Alternatively, a smaller write sizemay be chosen to ensure timely and safe storage of the data on SSD(i.e., smaller chunks written more frequently to SSD). Accordingly, thelog-structured layer of the file system may control write operations(i.e., chunks to SSD) both in frequency and size to sustain an effectivestreaming bandwidth to the SSDs of the storage array, therebyefficiently using the SSDs while overriding their garbage collection. Asnoted, a sufficient amount of write data over a contiguous range withina time frame (i.e., temporal locality) may be required to effectivelyoverride garbage collection in the SSDs.

Advantageously, the layered file system described herein providessequential log-structured layout of data and metadata as extents (i.e.,log-structured layout) on SSDs, embodied as flash storage devices, toreduce write amplification, while leveraging variable compression andvariable length extent features, as well as an extent de-duplicationfeature, of the storage I/O stack. The data may be organized as anarbitrary number of variable-length extents of one or more LUNs servedby the nodes, whereas the metadata may be organized as volume and extentmetadata. The volume and extent metadata may be separated and organizedas compact file system metadata structures residing in memories of thenodes to enable high performance processing of the extents with respectto, e.g., de-duplication and/or compression. In addition, the volume andextent store layer file systems may cooperate to store the extents onSSD in a log-structured manner that is flash friendly with respect torandom write operations and that reduces RAID-related writeamplification.

The foregoing description has been directed to specific embodiments. Itwill be apparent, however, that other variations and modifications maybe made to the described embodiments, with the attainment of some or allof their advantages. For instance, it is expressly contemplated that thecomponents and/or elements described herein can be implemented assoftware encoded on a tangible (non-transitory) computer-readable medium(e.g., disks and/or CDs) having program instructions executing on acomputer, hardware, firmware, or a combination thereof. Accordingly thisdescription is to be taken only by way of example and not to otherwiselimit the scope of the embodiments herein. Therefore, it is the objectof the appended claims to cover all such variations and modifications ascome within the true spirit and scope of the embodiments herein.

What is claimed is:
 1. A method comprising: organizing a storage arrayof solid state drives (SSDs) coupled to a node as at least one redundantarray of independent disks (RAID) group; organizing data and metadata onthe SSDs in a sequential log-structured layout, the data stored asextents of a logical unit (LUN) served by the node; gathering theextents to form at least one stripe written as a write operation thatspans all of the SSDs in the RAID group; and controlling a frequency anda size of the write operation to sustain a streaming bandwidth to theSSDs and to override garbage collection in the SSDs.
 2. The method ofclaim 1 wherein controlling the frequency and the size of the writeoperation further comprises varying the size of the write operationdepending on a write load served by the node.
 3. The method of claim 2wherein the size of the write operation varies from a minimum size to amaximum size depending on the write load, wherein the maximum size is atleast 10 times the minimum size.
 4. The method of claim 2 whereinvarying the size of the write operation comprises determining the sizeof the write operation based on pressure in a write-back cache of thenode.
 5. The method of claim 2 wherein controlling the frequency and thesize of the write operation further comprises waiting until enough datais accumulated in a write-back cache of the node to form a sufficientlylarge write size to override garbage collection in the SSDs.
 6. Themethod of claim 2 wherein controlling the frequency and the size of thewrite operation further comprises choosing a sufficiently small writesize to ensure timely storage of the data on the SSDs.
 7. The method ofclaim 1 wherein controlling the frequency and the size of the writeoperation comprises writing the extents of the write operation over asame SSD address range as previously written extents on the SSD tooverride the garbage collection in the SSDs.
 8. A system comprising: aprocessor connected to a memory; a storage array connected to theprocessor and having solid state drives (SSDs); a storage input/output(I/O) stack executing on the processor to: organize the SSDs as at leastone redundant array of independent disks (RAID) group; organize data andmetadata on the SSDs in a sequential log-structured layout, the datastored as extents of a logical unit (LUN) served by the storage I/Ostack; gather the extents to form at least one stripe written as a writeoperation that spans all of the SSDs in the RAID group; and control afrequency and a size of the write operation to sustain a streamingbandwidth to the SSDs and to override garbage collection in the SSDs 9.The system of claim 8 wherein the size of the write operation variesdepending on a write load served the storage I/O stack.
 10. The systemof claim 9 wherein the size of the write operation varies from a minimumsize to a maximum size depending on the write load, wherein the maximumsize at least 10 times the minimum size.
 11. The system of claim 9wherein the size of the write operation is determined based-on pressurein a write-back cache connected to the processor.
 12. The system ofclaim 9 further comprising a write-back cache connected to theprocessor, the write-back cache configured to accumulate enough data toform a sufficiently large write size to override garbage collection inthe SSDs.
 13. The system of claim 8 wherein the extents of the writeoperation are written over a same SSD address range as previouslywritten extents on the SSD to override the garbage collection in theSSDs.
 14. A non-transitory computer readable medium including programinstructions for execution on a processor, the program instructions whenexecuted operable to: organize a storage array of solid state drives(SSDs) coupled to a node as at least one redundant array of independentdisks (RAID) group; organize data and metadata on the SSDs in asequential log-structured layout, the data stored as extents of alogical unit (LUN) served by the node; gather the extents to form atleast one stripe written as a write operation that spans all of the SSDsin the RAID group; and control a frequency and a size of the writeoperation to sustain a streaming bandwidth to the SSDs and to overridegarbage collection in the SSDs.
 15. The non-transitory computer readablemedium of claim 14 wherein program instructions when executed arefurther operable to: vary the size of the write operation depending on awrite load served by the node.
 16. The non-transitory computer readablemedium of claim 15 wherein the size of the write operation varies from aminimum size to a maximum size depending on the write load, wherein themaximum size is at least 10 times the minimum size.
 17. Thenon-transitory computer readable medium of claim 15 wherein programinstructions when executed are further operable to: determine the sizeof the write operation based-on pressure in a write-back cache of thenode.
 18. The non-transitory computer readable medium of claim 15wherein program instructions when executed are further operable to: waituntil enough data is accumulated in a write-back cache of the node toform a sufficiently large write size to override garbage collection inthe SSDs.
 19. The non-transitory computer readable medium of claim 15wherein program instructions when executed are further operable to:choose a sufficiently small write size to ensure timely storage of thedata on the SSDs.
 20. The non-transitory computer readable medium ofclaim 14 wherein program instructions when executed are further operableto: write the extents of the write operation over a same SSD addressrange as previously written extents on the SSD to override the garbagecollection in the SSDs.